-
Notifications
You must be signed in to change notification settings - Fork 592
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor: use 1 for vnode count of singletons #18753
Conversation
This stack of pull requests is managed by Graphite. Learn more about stacking. Join @BugenZhao and the rest of your teammates on Graphite |
602ed69
to
9a2c6ce
Compare
This reveals a severe bug that we persist and notify about the internal tables before the vnode counts are correctly resolved, leading to inconsistency between the stream nodes and the catalog. |
c2f007c
to
7a83d1a
Compare
9a2c6ce
to
b427b00
Compare
Will first fix in #18759 |
b427b00
to
048b217
Compare
eb7d2cf
to
beba88d
Compare
048b217
to
5716e51
Compare
beba88d
to
23a1d96
Compare
5716e51
to
778ca89
Compare
778ca89
to
602c622
Compare
ccf26b4
to
8e6c87d
Compare
09fe68d
to
94dadd9
Compare
src/meta/model_v2/migration/src/m20240911_083152_variable_vnode_count.rs
Outdated
Show resolved
Hide resolved
8451b92
to
3e73343
Compare
This PR has been significantly reworked. Please review it again based on the updated PR body. 🥰 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we can get this PR into release 2.1, we don't need a new migration step but only to update this existing one.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Decide to include this PR in v2.2 instead. Switch to a new migration file.
Signed-off-by: Bugen Zhao <[email protected]>
Signed-off-by: Bugen Zhao <[email protected]>
Signed-off-by: Bugen Zhao <[email protected]>
Signed-off-by: Bugen Zhao <[email protected]>
Signed-off-by: Bugen Zhao <[email protected]>
Signed-off-by: Bugen Zhao <[email protected]>
3e73343
to
0a9cb10
Compare
Signed-off-by: Bugen Zhao <[email protected]>
Signed-off-by: Bugen Zhao <[email protected]>
Signed-off-by: Bugen Zhao <[email protected]>
…19151) Signed-off-by: Bugen Zhao <[email protected]>
…19151) Signed-off-by: Bugen Zhao <[email protected]>
…19151) Signed-off-by: Bugen Zhao <[email protected]>
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
This is a progress towards #15900.
Try to address #18515 (comment), i.e., use 1 for vnode count of singletons. There are several cases to consider:
Source
executor. In this case, we still treat it as singleton and assign it with 1 as the vnode count.vnode_count
of a state table is not always the same as the length of thevnode_bitmap
of its associated fragment. This should be correctly handled while performing batch scans over them.VnodeCountCompat::vnode_count
should return 1 for structures persisted before, if they are believed to be singleton.vnode_count
field of singletons. The logic here must be consistent with the one inVnodeCountCompat::vnode_count
, as under SQL backend, we still have some table catalogs persisted in form of protobuf, like the internal table catalogs in plan nodes.Checklist
./risedev check
(or alias,./risedev c
)Documentation
Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.